Network camera system and control method therefore

ABSTRACT

A network camera system is provided that detects an occurrence of a condition change at a periphery of an image pickup apparatus, obtains a direction of the detected condition change, and notifies a user of the occurrence of the condition change and a direction of the condition change on a display of a terminal device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a camera system including a camera, andin particular, relates to a network camera system that displays an imagephotographed by a camera using a terminal device connected via anetwork.

2. Description of the Related Art

Conventionally, there is a network camera system that displays an imagephotographed by a camera using a terminal device connected via anetwork. In addition, there is a network camera system that displays theimage on the terminal device while controlling an image taking directionof the camera through the terminal device. There is, for example, theWebView Livescope system developed by Canon, Inc. The system of thiskind is utilized in a monitoring camera and a video conference system.

In a conventional monitoring system, a method is employed in which theimage taking direction is previously set and the camera is automaticallyadjusted in the previously-set direction when a sensor reacts to theobject.

In addition, in a system such as the video conference system, a methodis employed in which a microphone determines a direction in which avoice comes. The image taking direction of the camera is automaticallyadjusted to the direction in which the voice comes (refer to JapanesePatent Application Laid-Open No. 07-284186).

However, in the conventional network camera system, when a condition ofimage taking changes in an area other than areas that are currentlybeing displayed on the terminal device, a user of the camera cannotreadily recognize where the condition change occurs.

In addition, if the image taking direction of the camera isautomatically adjusted or changed by the sensor or the voice asdescribed above while the user is adjusting and changing the imagetaking direction of the camera through the terminal device, aninterference occurs and the user cannot perform proper operation.

SUMMARY OF THE INVENTION

The present invention is directed to providing a system in which, when aphoto-taking condition changes, a user can readily recognize an areawhere the condition change is occurring.

According to at least one exemplary embodiment of the present invention,a network camera system is provided which is configured to display animage picked up by an image pickup apparatus on a terminal device via anetwork. The system includes a detection unit configured to detect anoccurrence of a condition change at a periphery of the image pickupapparatus; an obtaining unit configured to obtain a direction of thecondition change that is detected by the detection unit; and anotification unit configured to notify an occurrence of the conditionchange and the direction of the condition change on a display of theterminal device.

According to an aspect of the present embodiment, the terminal devicedisplays a predetermined icon or an image in a superposed manner so asto show the direction of the condition change. According to anotheraspect of the present embodiment, the terminal device displays the iconor the image in a superposed manner in a direction of the conditionchange relative to a center of a video display area of the terminaldevice. According to another aspect of the present invention, theterminal device displays scroll bars that represent an operational rangeand a current value with respect to panning and tilting of the imagepickup apparatus and the icon or the image at a position correspondingto a condition change area on the scroll bars.

According to yet another aspect of the present embodiment, the icon hasa shape that can specify the direction of the condition change.According to still another aspect of the present invention, wherein theicon or the image corresponds to a kind of the condition change.Moreover, according to another aspect of the present invention, aplurality of the icons or the images may be displayed in a superposedmanner in a number of the condition changes.

According to yet another aspect of the present embodiment, the icon orthe image is not displayed in the superposed manner if an image of thecondition change area is included in a video display area of theterminal device. Furthermore, in another aspect of the presentinvention, while displaying the icon or the image in the superposedmanner, a size of the icon or the image can be changed. Additionally, inanother aspect of the present embodiment, the image pickup apparatusdisplays an image picked up after changing a shooting direction on theterminal device via the network, and the obtaining unit obtains thedirection of the condition change relative to the shooting direction.

According to yet another aspect of the present embodiment, the detectionunit comprises at least one of an optical sensor, a voice sensor, and atemperature sensor or a combination thereof. And moreover, according toanother aspect of the present invention, the image of the conditionchange area is cut off from an image of all areas previously picked upand stored that can be photographed by the image pickup apparatus. And,according to another aspect of the present invention, the terminaldevice may be configured to control the image pickup apparatus via thenetwork and change the shooting direction of the image pickup apparatusto the direction of the condition change by selecting the icon or theimage.

According to yet another aspect of the present embodiment, the imagepickup apparatus cuts off an arbitrary area of the photographed imageand displays the cut off area on the terminal device via the network,wherein the obtaining unit obtains the direction of the condition changerelative to the cut off area. According to yet another aspect of thepresent invention, the detection unit includes one of an optical sensor,a voice sensor, a temperature sensor, or an image analysis unit or acombination thereof.

And, according to yet another aspect of the present embodiment, theimage is the cut off image of the condition change area. Moreover,according to another aspect of the present invention, the imagedisplayed on the terminal device can be changed to the cut off an imageof the area that includes the condition change area by selecting theicon or the image.

According to at least one exemplary embodiment of the present invention,a method is provided for controlling a network camera system configuredto display an image picked up by an image pickup apparatus on a terminaldevice via a network. The method includes detecting an occurrence of acondition change at a periphery of the image pickup apparatus; obtaininga direction of the condition change that is detected by the detectionunit; and notifying an occurrence of the condition change and thedirection of the condition change on a display of the terminal device.

According to yet another exemplary embodiment of the present invention,an image pickup apparatus is provided which is adapted to send aphotographed image to a terminal device via a network. The apparatusincludes a detection unit configured to detect an occurrence of acondition change at a periphery of the image pickup apparatus; anobtaining unit configured to obtain a direction of the condition changethat is detected by the detection unit or information for obtaining thedirection of the condition change; and a transmission unit configured totransmit to the terminal device an occurrence of the condition changeand the direction of the condition change or the information forobtaining the direction of the condition change.

According to still another exemplary embodiment of the presentinvention, a terminal device is provided which is adapted to receive anddisplay an image picked up by an image pickup apparatus via a network.The device includes an obtaining unit configured to obtain a directionof a condition change at a periphery of the image pickup apparatus; anda notification unit configured to notify an occurrence of the conditionchange and the direction of the condition change on a display of theterminal device.

According to still another exemplary embodiment of the presentinvention, a computer readable medium is provided which containscomputer-executable instructions, the medium utilizable in an imagepickup apparatus adapted to transmit a photographed image to a terminaldevice via a network, the medium configured to cause the terminal deviceto perform processing. The medium includes computer-executableinstructions for detecting an occurrence of a condition change at aperiphery of the image pickup apparatus; computer-executableinstructions for obtaining a direction of the condition change that isdetected or information for obtaining the direction of the conditionchange; and computer-executable instructions for transmitting to theterminal device an occurrence of the condition change and the directionof the condition change or the information for obtaining the directionof the condition change.

Additionally, according to still another exemplary embodiment of thepresent embodiment, a computer readable medium is provided whichcontains computer-executable instructions, the medium utilizable in aterminal device configured to receive and display an image picked up byan image pickup apparatus via a network. The medium includescomputer-executable instructions for obtaining a direction of acondition change at a periphery of the image pickup apparatus; andcomputer-executable instructions for notifying an occurrence of thecondition change and the direction of the condition change on a displayof the terminal device.

Moreover, according to another exemplary embodiment of the presentinvention, an image taking apparatus is provided which includes an imagetaking unit configured to take an image and output an image signal ofthe image; a sensor unit configured to detect information about an areawider than an image angle of the image taking unit; an extraction unitconfigured to extract a subject candidate at least out of theinformation obtained by the sensor unit; and a determination unitconfigured to determine whether the extracted subject candidate is in orout of an image taking area of the image taking unit based on theinformation from the sensor unit and an image angle of the image takingunit.

According to yet another aspect of the present embodiment, theextraction unit extracts the subject candidate by further using theimage signal output from the image taking unit. In another aspect, theimage taking apparatus may further include a notification unitconfigured to notify a result determined by the determination unit.

According to still another aspect of the present embodiment, the imagetaking apparatus may further include a moving member detecting unitconfigured to detect whether the subject candidate is a moving member,and if it is determined that the subject candidate is a moving member,output movement anticipation information; wherein the notification unitfurther notifies the movement anticipation information; wherein themovement anticipation information includes at least one of informationabout a movement speed and direction of the subject candidate, timeuntil the subject candidate is framed-out to the outside of the imagetaking area if the subject candidate is in the image taking area of theimage taking unit, and time until the subject candidate is framed-in tothe inside of the image taking area if the subject candidate is outsideof the image taking area of the image taking apparatus.

According to another aspect of the present embodiment, the notificationunit is a display unit, and indicates the subject candidate with amarker. Also, according to another aspect, the notification unit is asound output unit and notifies the determined result by sound. Stillfurther, according to another aspect of the present embodiment, if thesubject candidate is out of the image taking area of the image takingunit, the notification unit notifies a positional relationship betweenthe image taking unit and the image taking area.

According to still another aspect of the present embodiment, the imagetaking apparatus may further include a control unit configured tocontrol the image taking unit so that the subject candidate enters theimage taking area of the image taking apparatus if it is determined thatthe subject candidate is out of the image taking area. According toanother aspect of the present embodiment, wherein the sensor unit is oneof an image taking unit, a sound receiving unit, a temperature sensor,or an infrared sensor. Moreover, according the image taking unit and thesensor unit are configured to select an output area of an image signalfrom the same image taking element.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a schematic diagram that illustrates an exemplaryconfiguration of network camera system according to a first embodimentof the present invention.

FIG. 2 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to the firstembodiment of the present invention.

FIG. 3 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to a fourthembodiment of the present invention.

FIG. 4 is a schematic diagram that illustrates an exemplaryconfiguration of a network camera system according to a sixth embodimentof the present invention.

FIG. 5 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to a sixthembodiment of the present invention.

FIG. 6 is a flow chart of exemplary processing performed in transmittingan image from a camera device according to the sixth embodiment of thepresent invention.

FIG. 7 is a flowchart of exemplary processing in which the terminaldevice receives and displays the image according to the sixth embodimentof the present invention.

FIGS. 8A and 8B are diagrams that illustrate an exemplary method ofcomputing a position of an image which shows an area where a conditionchange occurs, according to a seventh embodiment of the presentinvention.

FIG. 9 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to an eighthembodiment of the present invention.

FIG. 10 is a schematic diagram that illustrates a positionalrelationship of areas for an omnidirectional image according to a ninthembodiment of the present invention.

FIG. 11 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to aneleventh embodiment of the present invention.

FIG. 12 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to theeleventh embodiment of the present invention.

FIG. 13 is a schematic diagram that illustrates an example of a windowdisplayed by a display unit of a terminal device according to a twelfthembodiment of the present invention.

FIG. 14 is a block diagram that illustrates an exemplary functionaldiagram of a camera device according to a thirteenth embodiment of thepresent invention.

FIG. 15 is a flow chart of exemplary subject candidate detectionprocessing according to the thirteenth embodiment of the presentinvention.

FIG. 16 is a block diagram that illustrates an exemplary configurationof a subject candidate extraction unit according to the thirteenthembodiment of the present invention.

FIG. 17 is a flow chart of exemplary subject candidate extractionprocessing performed in the subject candidate extraction unit accordingto the thirteenth embodiment of the present invention.

FIG. 18 is a block diagram that illustrates an exemplary configurationof a material detection unit according to the thirteenth embodiment ofthe present invention.

FIG. 19 illustrates an example of a status class stored in a statusdetection unit according to the thirteenth embodiment of the presentinvention.

FIG. 20 illustrates an exemplary configuration of an individualrecognition unit according to the thirteenth embodiment of the presentinvention.

FIG. 21 illustrates an example of a display in a case where a subjectcandidate is in an image display unit according to the thirteenthembodiment of the present invention.

FIG. 22 illustrates an example of a display in a case where a subjectcandidate is outside of an image display unit according to thethirteenth embodiment of the present invention.

FIG. 23 illustrates a view for explaining an image taking unit having afunction of a wide range sensor according to a modification of thethirteenth embodiment of the present invention.

FIG. 24 is a flow chart of exemplary image taking processing accordingto a modification of the thirteenth embodiment of the present invention.

FIG. 25 is a flow chart of exemplary image processing according to amodification of the thirteenth embodiment of the present invention.

FIG. 26 is a block diagram that illustrates an exemplary functionalconfiguration of a camera device according to a fourteenth embodiment ofthe present invention.

FIG. 27 is a flow chart of exemplary subject candidate detectionprocessing according to the fourteenth embodiment of the presentinvention.

FIG. 28 is a block diagram that illustrates an exemplary configurationof a moving member anticipation unit according to the fourteenthembodiment of the present invention.

FIG. 29 is a flow chart of an exemplary moving member anticipationprocessing according to the fourteenth embodiment of the presentinvention.

FIG. 30 illustrates an example of a display of an anticipated result inthe moving member anticipation unit in a case where a subject candidateis in an image display unit according to the fourteenth embodiment ofthe present invention.

FIG. 31 illustrates an example of a display of an anticipated result inthe moving member anticipation unit in a case where a subject candidateis out of an image display unit according to the fourteenth embodimentof the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments, features and aspects of the present inventionwill now be described in detail with reference to the drawings. It isnoted that the relative arrangement of the components, the numericalexpressions and numerical values set forth in these embodiments are notintended to limit the scope of the present invention unless specificallystated otherwise.

First Exemplary Embodiment

FIG. 1 is a schematic diagram that illustrates an exemplaryconfiguration of a network camera system according to a first embodimentof the present invention. The network camera system is constituted by acamera device 1, a network 2, and a terminal device 3.

The camera device 1 includes an image pickup unit 11 that picks up animage, a camera control unit 12 that controls a direction of imagetaking, a zoom value, and an image taking processing, and a sensor unit13 that detects a change of an environment in which the image taking isperformed, based on a condition of light, a voice, a temperature, andthe like. Further, the camera device 1 includes a control unit 15 thatcontrols the entire camera device 1 and a network interface 14 thatsends and receives data via the network 2. It is noted that the cameradevice 1 can change a shooting direction, which is, for example, of aslew type.

The terminal device 3 includes a display unit 33 that displays theimage, an operation input unit 34 that receives an operation performedby a user, a control unit 32 that exercises control over the entireapparatus, and a network interface 31 that sends and receives data viathe network 2.

The network camera system that has such a constitution can send theimage data picked up by the image pickup unit 11 of the camera device 1to the terminal device 3 and display the image on the display unit 33.

In addition, the network camera system that has such a constitutionsends information of the operation performed through the operation inputunit 34 of the terminal device 3 to the camera device 1, and, inresponse to the information, the camera control unit 12 changes andadjusts the shooting direction, the zoom value, and image capturingoperations performed by the image pickup unit 11.

In this case, when the sensor unit 13 of the camera device 1 detects achange of the environment in which the image taking is performed and aposition where the change occurs, the control unit 15 seeks a directionwhere the condition changes, relative to the shooting directioncontrolled by the camera control unit 12. Then, the control unit 15sends information about the obtained direction where the conditionchanges, to the terminal device 3, while a notification of the conditionchange is issued.

FIG. 2 illustrates an example of a window displayed by the display unit33 of the terminal device 3 according to the first embodiment of thepresent invention. Reference numeral 4 denotes a video display area inwhich the image picked up by the image pickup unit 11 of the cameradevice 1 is displayed, and reference numeral 41 denotes a center of thevideo display area that is a center (i.e., the shooting direction) ofthe video display area 4. Reference numeral 42 denotes a notificationicon that is displayed and superposed in the video display area 4 so asto notify the occurrence of the condition change and the direction ofthe condition change.

As shown in FIG. 2, the notification icon 42 is displayed in such amanner that the direction of the condition change is indicated. Thedisplay area center 41 serves as its reference. For example, the videodisplay area 4 is divided and the notification icon 42 is displayed in asegmented area existing in the direction of the condition change whilethe display area center 41 (i.e., the shooting direction) serves as thereference. In this case, even when the video display area 4 is notdivided, the notification icon 42 may take a shape that can specify andindicate the direction of the condition change, for example, anarrow-like shape that is oriented in the direction of the conditionchange.

In addition, the notification icon 42 may be configured to correspond toa kind of the condition change, for example, a kind of the sensor unit13 (a light sensor, a voice sensor, a temperature sensor, and the like)that has detected the occurrence of the condition change.

The terminal device 3, when an operation for selecting the notificationicon 42 is performed through the display unit 33 via the operation inputunit 34, sends information of the operation to the camera device 1. Inresponse to this information, the camera control unit 12 of the cameradevice 1 changes and adjusts the image pickup direction to the directionof the condition change. That is, the notification icon 42 serves as atrigger for changing and adjusting the image pickup direction of thecamera device 1.

Second Exemplary Embodiment

In the first embodiment, the notification icon 42 is displayed in asuperposed manner. However, the embodiment of the present invention isnot limited to this constitution. Instead of the notification icon 42,the images of all areas that can be photographed by the camera device 1may be stored in advance, and the image obtained by cutting off the areawhere the condition change occurs may be displayed.

Third Exemplary Embodiment

In the first embodiment and the second embodiment, the control unit 15of the camera device 1 computes the direction of the condition change.However, the embodiment of the present invention is not limited to thisconstitution. Instead of the control unit 15, the terminal device 3 maycompute the direction of the condition change. To be more specific, thesensor unit 13 of the camera device 1 detects the occurrence of thecondition change and the position of the condition change in itsperipheral. The sensor unit 13 notifies the occurrence of the conditionchange and the position where the condition changes, to the terminaldevice 3. Upon receipt of the notification of the information, theterminal device 3 computes the direction of the condition changerelative to the shooting direction controlled by the camera control unit12.

Fourth Exemplary Embodiment

In the first, the second, and the third embodiment, only the directionof the condition change is computed. However, the constitution of thepresent invention is not limited to this. That is, the zoom valuesuitable to capture the image of the area where the condition changesmay be computed. To be more specific, when the sensor unit 13 of thecamera device 1 detects the occurrence of the condition change and theposition where the condition change occurs in its peripheral, a distancefrom the camera device 1 to the position where the condition changeoccurs and a range of the condition change are computed if it ispossible depending on the type of the sensor unit 13. Then, the sensorunit 13 sends the information about the position of the conditionchange, the distance to the position of the condition change, and therange of the condition change to the terminal device 3, as well as theoccurrence of the condition change to the terminal device 3. When thenotification of the information is received, the terminal device 3computes the direction of the condition relative to the shootingdirection controlled by the camera control unit 12. In addition, theterminal device 3 computes the zoom value based on the receivedinformation of the distance to the position of the condition change andthe range of the condition change. If the zoom value cannot be obtainedby the computation, a previously-set zoom value or the current zoomvalue may be used as the zoom value.

As an example of the display by the display unit 33 of the terminaldevice 3, the notification icon 42 can be displayed as shown in FIG. 2in proportion to a difference between the current zoom value and thecomputed zoom value. Namely, the notification icon 42 of a small sizecan be used if the computed zoom value is included in a telephoto sideand the icon of a large size can be used if the computed zoom value isincluded in a wide angle side. Alternatively, the notification icon 42can be displayed in a large size if the computed zoom value is includedin the telephoto side and in a small size if the computed zoom value isincluded in the wide angle side.

When the notification icon 42 is selected through the display unit 33via the operation input unit 34, the terminal device 3 sends informationof the operation to the camera device 1. In response to the information,the camera control unit 12 of the camera device 1 changes and adjuststhe shooting direction to the direction of the condition change, andchanges the zoom value from the current zoom value to the computed zoomvalue. That is, the notification icon 42 serves as a trigger forchanging and adjusting the image pickup direction of the camera device 1and for changing the zoom value.

Fifth Exemplary Embodiment

In a fifth embodiment of the present invention, the display by thedisplay unit 33 of the terminal device 3 is altered. FIG. 3 illustratesan example of the display by the display unit 33 of the terminal device3. Reference numeral 4 denotes a video display area in which the imagepicked up by the image pickup unit 11 of the camera device 1 isdisplayed.

Reference numeral 43 denotes a pan scroll bar that represents anoperational range in a direction of panning. Reference numeral 431denotes a pan current value that represents in the pan scroll bar 43 apan position where the image taking is currently performed, andreference numeral 432 denotes a pan notification icon that represents inthe pan scroll bar 43 a position corresponding to the area where thecondition changes.

Reference numeral 44 denotes a tilt scroll bar that represents anoperational range in a tilt direction, and reference numeral 441 denotesa tilt current value that represents in the tilt scroll bar 44 a tiltposition where the image taking is currently performed. Referencenumeral 442 denotes a tilt notification icon that represents in the tiltscroll bar 44 a position corresponding to the area where the conditionchanges.

In addition, reference numeral 45 denotes a zoom scroll bar thatrepresents an operational range of the zoom, and reference numeral 451denotes a zoom current value that represents in the zoom scroll bar 45 acurrent zoom value. Reference numeral 452 denotes a zoom notificationicon that represents in the zoom scroll bar 45 a zoom value that issuitable to pick up the image of the area where the condition changes.

The sensor unit 13 of the camera device 1 detects the occurrence of thecondition change and the position of the condition change in itsperipheral. The sensor unit 13 notifies the occurrence of the conditionchange and the position where the condition change occurs, to theterminal device 3. In response to the notification, the terminal device3 computes the direction of the condition change relative to theshooting direction controlled by the camera control unit 12, andcomputes a pan value and a tilt value to pick up the image of the areawhere the condition changes. Then the terminal device 3 displays thenotification icons 432 and 442 on the scroll bars 43 and 44. In a casewhere the zoom value is computed as described in the fourth embodiment,the notification icon 452 is displayed on the scroll bar 45.

When an operation for selecting the pan notification icon 432, the tiltnotification icon 442, or the zoom notification icon 452 is performedthrough the display unit 33 via the operation input unit 34, theterminal device 3 sends information of the operation to the cameradevice 1. In response to the information, the camera control unit 12 ofthe camera device 1 changes and adjusts the image pickup direction tothe direction of the condition change. In addition, the camera controlunit 12 changes the zoom value from the current zoom value to thecomputed zoom value. That is, the pan notification icon 432 and the tiltnotification icon 442 serve as triggers for changing and adjusting theimage pickup direction of the camera device 1, and the zoom notificationicon 452 serves as a trigger for changing the zoom value.

Sixth Exemplary Embodiment

FIG. 4 is a schematic diagram that illustrates an exemplaryconfiguration of a network camera system according to a sixth embodimentof the present invention. The network camera system is constituted bythe camera device 1 capable of a wide-angle shot, the network 2, and theterminal device 3.

With respect to a constitution of the camera device 1, and inparticular, components that have not yet been described with referenceto FIG. 1, reference numeral 16 denotes a wide angle image pickup unitcapable of picking up the image with a wide angle by using a fisheyelens, a rotational symmetrical mirror, and the like. Reference numeral17 denotes an image cutting off unit that cuts off the area to betransmitted to the terminal device 3, out of the image picked up by thewide-angle image pickup unit 16. Reference numeral 18 denotes an imageanalysis unit that detects a change in the condition such as a movement,a person (that is, a face or a whole part of a person), and a color, byanalyzing the image picked up by the wide-angle image pickup unit 16.

As to a constitution of the terminal device 3, an in particular,components that have not yet been described with reference to FIG. 1,reference numeral 35 denotes an image combining unit that superposes theimage to be displayed on the display unit 33 with other images.

In the network system having such a configuration, the area specifiedthrough the terminal device 3 can be cut off out of the image picked upby the wide-angle image pickup unit 16 of the camera device 1 using theimage cut off unit 17. The cut off image is sent to the terminal device3, and displayed on the display unit 33. That is, the camera device 1can change the area of the image to be cut off by the image cut off unit17 in accordance with the operation performed through the operationinput unit 34 of the terminal device 3. The cut off area is sent to theterminal device 3. The received image serves as a main image.

When the condition change of the movement, the person, and the color andthe position of the condition change is detected by the image analysisunit 18 of the camera device 1, the image cut off unit 17 cuts off thearea where the condition changes, out of the image taken by thewide-angle image pickup unit 16. Then, the information about the cut offarea where the condition changes s is sent to the terminal device 3together with the main image. Thus, the image obtained by cutting offthe condition change area can be superposed onto the main image anddisplayed on the display unit 33 of the terminal device 3.

When the sensor unit 13 of the camera device 1 detects the occurrence ofthe condition change in its peripheral and the position of the conditionchange, the image cut off unit 17 also cuts off the condition changearea out of the image taken by the wide-angle image pickup unit 16, andthe cut off image is sent to the terminal device 3 together with themain image. Thus, the image obtained by cutting off the condition changearea can be superposed onto the main image and displayed on the displayunit 33 of the terminal device 3.

FIG. 5 is a schematic diagram that illustrates an exemplary windowdisplayed by the display unit 33 of the terminal device 3. Referencenumeral 4 denotes a video display area in which the main image isdisplayed; reference numeral 41 denotes a center of the display areathat is a center of the video display area 4; and reference numeral 48denotes a notification image that is superimposed and displayed in thevideo display area 4 to notify the occurrence of the condition changeand the direction of the condition change. The image is obtained bycutting off the condition change area as described above.

As shown in FIG. 5, the notification image 48 is displayed in such amanner that the direction of the condition change is indicated while thedisplay area center 41 serves as the reference. For example, the videodisplay area 4 is divided and is displayed in a divided area existing inthe direction of the condition change relative to the display areacenter 41.

In addition, the notification image 48 can be formed as an image thatcorresponds to a kind of the condition change, for example, a kind ofthe sensor unit 13 (a light sensor, a voice sensor, a temperaturesensor, and the like) that has detected the occurrence of the conditionchange. Further, the notification image 48 may be an image thatcorresponds to the kind of the movement, person, and color analyzed bythe image analysis unit 18 that has detected the condition change.

When the notification image 48 is selected through the display unit 33via the operation input unit 34, the terminal device 3 sends informationof the selection to the camera device 1. In response to the information,the camera control unit 12 of the camera device 1 changes the imagedisplayed by the terminal device 3 from the main image to the imageobtained by cutting off the area which includes the condition changearea. That is, the notification image 48 serves as a trigger forchanging the image displayed by the terminal device 3 from the mainimage to the image obtained by cutting off the area including thecondition change area.

Hereinbelow, exemplary processing in the network camera system accordingto the sixth embodiment is described in further detail. FIG. 6 is a flowchart of a processing in transmitting the image from a camera device 1.FIG. 7 is a flow chart of a processing in which the terminal device 3receives and displays the image.

The processing shown in FIG. 6 is executed in parallel with theprocessing in which the image cut off unit 17 cuts off the areaspecified through the terminal device 3 out of the image picked up bythe wide-angle image pickup unit 16 of the camera device 1 and sends thedata of the cut off area to the terminal device 3.

Now referring to FIG. 6, first, the image analysis unit 18 obtains alatest image from the wide-angle image pickup unit 16 (step S01). Theimage obtained here is a wide-angle image picked up by the wide-angleimage pickup unit 16. Next, a determination is made as to whether awide-angle image for comparison obtained just before in step S01, existsin a buffer of the image analysis unit 18 (step S02).

If it is determined that there exists the wide-angle image in thebuffer, the wide-angle image in the buffer and the wide-angle imageobtained in step S01 are analyzed and compared to each other to computean amount of change (step S03). For carrying out the analysis and thecomparison, for example, it is considered whether the variance occursbetween images, pixel by pixel, or block by block, to where a similarpattern has been displaced between images, whether there exists apattern that matches a face of a person. In addition, it can beconsidered whether the face of the person has been displaced betweenimages, whether there exists the area that matches a specific color, andwhether the specific color has been displaced between images. The amountof change is computed by performing the above method of analyzing andcomparing or by combining two or more of such methods, so that thewide-angle images are mutually compared.

Then, a determination is made as to whether a condition change hasoccurred, based on whether the amount of change obtained by thecomparison between the wide-angle images exceeds a threshold valuepreviously set per each analysis method or the threshold previously setby combining the analysis methods (step S04). If it is determined thatthe condition change has occurred, the information about the conditionchange area is sent from the image analysis unit 18 to the image cut offunit 17. Then, the image cut off unit 17 cuts off the condition changearea out of the wide-angle image, and the data of the cut off area issent to the terminal device 3 (step S05).

After step S05, or if the wide-angle image has not been stored in thebuffer in step S02, or if there is no variance between the wide-angleimages in step S04, the wide-angle image obtained in step S01 is storedin the buffer of the image analysis unit 18 (step S06). If only oneimage is to be stored in the buffer, the image is stored by overwriting.If a plurality of images is to be stored in the buffer, the plurality ofimages is stored while oldest images are deleted.

The exemplary processing shown in FIG. 7 is performed when the terminaldevice 3 receives the main image from the camera device 1, displays thereceived main image on the display unit 33, and when the image of thecondition change area which is cut off in step S05 as shown in FIG. 6 issent to the terminal device 3.

Referring to FIG. 7, first, when the camera device 1 receives the cutoff image of the condition change area from the camera device 1, themain image area of the wide-angle image and the condition change area iscomputed (step S07). Next, a ratio of the condition change area to themain image area is computed. If the ratio exceeds a predeterminedthreshold value, it is determined that the condition change area existsin the main image (step S08), and in this case, the main image isdisplayed as it is (step S11). This is because, if the main imageincludes the condition change area, the cut off image of the conditionchange area does not need to be superposed on the main image.

If it is determined that the condition change area is not included inthe main image area (step S08), a vector oriented from the center of themain image area to the center of the condition change area is computed(step S09). Then, the position and the size of the image whose conditionchange area is cut off and superposed onto the main image, are computedbased on the computed vector. The cut off image is superposed (stepS10), and the resultant image is displayed on the display unit 33 (stepS11). AS an example of computing the position of the cut off image ofthe condition change area, a method may be employed such that a positiongoing from the center of the main image in a vector direction and at aconstant distance from an edge of the main image may be defined as asuperposing position. In addition, as a method of computing the size ofthe cut off image of the condition change area, a method can be employedsuch that the size of the image is computed by multiplying the conditionchange area by the ratio of the condition change area to the main image,and also by a coefficient that is in an inverse proportion or a directproportion to the size of the vector. The coefficient, for example, canbe determined such that the superposing area to be small when the vectoris large (when the distance is large) and allows the superposing area tobe large when the vector is small (when the distance is small).

It is also noted that in the present embodiment, the icon as describedin the first embodiment may be utilized instead of the notificationimage 48.

Seventh Exemplary Embodiment

In a seventh embodiment, as an example, a method of computing theposition of the cut off image of the condition change area andsuperposing onto the main image is described which is shown in step S10of the flow chart in FIG. 7. FIG. 8A is a diagram that illustrates adistribution of angles of vectors oriented from the main image areacenter to the condition change area center. FIG. 8B is a diagram thatillustrates on which portion of the video display area 4, the conditionchange area is superposed based on the distribution of the angles of thevectors.

If the distribution is determined to be the angle “1” in FIG. 8A, thecondition change area is superposed onto the video display area “I” inFIG. 8B. In this way, “2” is superposed onto “II”, “3” onto “III”, “4”onto “IV”, “5” onto “V”, “6” onto “VI”, “7” onto “VII”, and “8” onto“VIII”.

As a method of distributing the angles as shown in FIG. 8A, the anglesare distributed evenly by 45 degrees. In addition, an aspect ratio ofthe video display area 4 is 4:3, and therefore are two distributionmethods of the angles shown in FIG. 8A that are calculated byapproximation. In each of the two methods, the angles are respectivelyset to an angle (1) an and angle (2) as shown in Table 1 below.

TABLE 1 Distribution Angle (1) Angle (2) 1 3 × 60/7 3 × 180/17 2 60 5 ×180/17 3 4 × 60/7 4 × 180/17 4 60 5 × 180/17 5 3 × 60/7 3 × 180/17 6 605 × 180/17 7 4 × 60/7 4 × 180/17 8 60 5 × 180/17

Eight Exemplary Embodiment

In an eighth embodiment of the present invention, the display on thedisplay unit 33 of the terminal device 3 is altered. FIG. 9 illustratesa window displayed by the display unit 33 of the terminal device 3.Reference numeral 4 denotes a video display area in which the main imageis displayed. Reference numeral 43 donates a pan scroll bar thatrepresents an operational range in a pan direction, and referencenumeral 431 donates a pan current value that represents in the panscroll bar 43 a pan position that is currently cut off as the mainimage. Reference numeral 433 denotes a pan notification image thatrepresents in the pan scroll bar 43 a portion corresponding to thecondition change area.

Reference numeral 44 denotes a tilt scroll bar that represents anoperational range in a tilt direction, and reference numeral 441 denotesa tilt current value that represents in the tilt scroll bar 44 a tiltposition currently cut off as the main image. Reference numeral 443denotes a tilt notification image that represents in the tilt scroll bar44 a position corresponding to the condition change area.

Reference numeral 45 denotes a zoom scroll bar that represents anoperational range of the zoom, and reference numeral 451 denotes a zoomcurrent value that represents in the zoom scroll bar 45 a current zoomvalue. Reference numeral 453 denotes a zoom notification image thatrepresents in the zoom scroll bar 45 a zoom value that is suitable topick up the image of the condition change area.

The sensor unit 13 and the image analysis unit 18 of the camera device 1detect the condition change and the position of the condition change. Inthat case, the image of the condition change area cut off by the imagecut off unit 17 is sent to the terminal device 3, together with the mainimage. In response to the information, the terminal device 3 computes apan value, a tilt value, and a zoo value so that the condition changeposition becomes the main image, and displays the cut off image of thecondition change area, as the pan notification image 483, the tiltnotification image 443, and the zoom notification image 453respectively. In this embodiment, the cut off image of the conditionchange area is displayed with respect to each of the pan notificationimage 483, the tilt notification image 443, and the zoom notificationimage 453. However, the present invention is not limited to thisembodiment. That is, the cut off image of the condition change area canbe displayed for either one of the pan notification image 483, the tiltnotification image 443, or the zoom notification image 453, and otherimage is displayed for the other notification images. Otherwise, theicon can be displayed instead of the image.

When an operation for selecting the pan notification image 483, the tiltnotification image 443, or the zoom notification image 453 is performedthrough the display unit 33 via the operation input unit 34, theterminal device 3 sends information of the operation to the cameradevice 1. In response to the information, the camera device 1 changesthe image displayed by the terminal device 3 from the main image to theimage including the condition change area. That is, the pan notificationimage 483 and the tilt notification image 443 serve as triggers forchanging the image displayed by the terminal device 3 to the imageincluding the condition change area, and the zoom notification image 453serves as a trigger for changing the zoom value.

Ninth Exemplary Embodiment

The camera device 1 described in the sixth embodiment may be anomnidirectional camera whose wide-angle image pickup unit 16 has avisual field area of 360 degrees at a maximum. FIG. 10 is a schematicdiagram that illustrates a positional relationship between areas.Reference numeral 61 denotes all areas of an omnidirectional imagepicked up by the wide-angle image pickup unit 16. Reference numeral 62denotes an image analysis unit 18 that is the condition change area ofthe movement, the person, the color and the like. Reference numeral 63denotes the area of the main image specified through the terminal device3.

In FIG. 10, when the positional relationship of the condition changearea 62 is measured relatively with the area of the main image 63serving as the center, the condition change area 62 is positioned to theright of the main image area 63. That is, the condition change area 62is positioned closer to the circumference than the main image area 63.Further, in a case of the omnidirectional camera in which thecircumference is considered to be a underside, the condition changeoccurs in a lower right direction of the area cut off as the main image.The image combining unit 35 superposes the cut off image of thecondition change area 62 (notification image 48) onto the main image,based on a direction where the condition change occurs relative to themain image area 63.

Tenth Exemplary Embodiment

If a plurality of terminal devices 3 is connected to the camera device1, the control unit 15 manages the main image area specified througheach of the plurality of terminal devices 3, and the image cut off unit17 cuts off the main image with respect to each terminal device 3. Whenthe sensor unit 13 and the image analysis unit 18 detects the conditionchange and the position of the condition change, the image cut off unit17 cuts off the image of the condition change and the data of the cutoff image is sent to each of the terminal devices 3. In each of theterminal devices 3, the image combining unit 35 superposes the cut offimage of the condition change area (the notification image 48) onto themain image and displays the superposed image on each image display unit33.

In this case, the image can be superposed and displayed only on some ofthe terminal devices 3, instead of superposing and displaying the cutoff image of the condition change area onto the main image in all theterminal devices 3. For example, if the condition change area isincluded in the area of at least one main image, a transmission of thecut off image of the condition change area to the terminal device 3receiving the main image is suspended. This is because the terminaldevice 3 that displays the main image needs not superpose thenotification image 48 if the main image includes the condition changearea.

Further, the terminal device 3 can be selected in which main image isthe area that is the closest to the condition change area in thewide-angle image (or the omnidirectional image). The notification image48 is superposed only on the selected terminal device 3.

Eleventh Exemplary Embodiment

If the sensor unit 13 and the image analysis unit 18 have detected aplurality of condition changes, a plurality of notification images 48(or the notification icons 42) that corresponds to each conditionalchange can be superposed and displayed.

FIG. 11 is a window displayed by the display unit 33 on the terminaldevice 3. There arises a case where, while a first condition changeoccurs and a first notification image 48 (1) is being displayed, a nextcondition change is detected. In this case, a notification number 47 (1)is displayed in the vicinity of the first notification image 48 (1).Additionally, a notification image 48 (2) and a notification number 47(2) of the nest condition change are displayed. The notification numbersare provided with a unique number in an ascending order from earliernotification, or otherwise, in an order from later notification.

In the same way, when a third condition change or beyond the third isnotified, the notification image 48 and the notification number 47 aremutually superposed and displayed. If the plurality of notificationimages 48 is overlapped and displayed, the notification image 48 of anearlier notification may be displayed on the front, or otherwise, thenotification image 48 of a later notification can be displayed on thefront. Or alternatively, notification image 48 of a higher priority thatis specified in advance can be displayed on the front. Such priority canbe determined depending on the kind of the sensor that has detected thecondition change, the kind of the image analysis, and the range of thedetection.

When the condition change ends or a given period of time has elapsed,the notification image 48 is cleared. At that stage, the notificationnumber 47 related to the notification image 48 is cleared at the sametime. In addition, when the notification image 48 and the notificationnumber 47 are cleared, the number of remaining notification image 47 isupdated. When the notification image and the notification number 47 arecleared and if there is only one notification image 48 left, thenotification number 47 related to the remaining notification image 48 isalso cleared.

FIG. 12 is a window displayed on the display unit 33 of the terminaldevice 3. When a next condition change is detected while a firstcondition change occurs and a first notification image 48 (1) isdisplayed, a second notification image 48 (2) is displayed in a sizesmaller than the size of the notification image 48 (2) displayed in anordinary case. If a third condition change or beyond the third isdetected, the notification image 48 (2) is displayed by a largerreduction ratio by which the notification image 48 (2) is displayedsmaller than a previous notification image 48.

On the other hand, conversely, when the first notification image 48 (1)is displayed and if a next condition change is detected in the meantime, the first notification image 48 (1) can be displayed in a sizesmaller than display in an ordinary case, and the second notificationimage 48 (2) can be displayed in the size of the ordinary case. If athird condition change or beyond the third is detected, the notificationimage 48 that has been displayed so far is further reduced from the sizeof display until then. The notification image 48 that is notified latestis displayed in the ordinary size, and the notification images that havebeen notified before the latest-notified notification image 48 aredisplayed in smaller sizes than before at a reduction ratio thatincreases gradually in a notification order.

If the plurality of displayed notification images 48 are overlapped, thenotification image 48 of an earlier notification may be displayed on thefront, or otherwise, the notification image 48 of a later notificationcan be displayed on the front. Or alternatively, the notification image48 of a higher priority that is specified in advance can be displayed onthe front. Such priority can be determined depending on the kind of thesensor that has detected the condition change, the kind of the imageanalysis, and the range of the detection.

When the condition change ends or a given period of time has elapsed,the notification image 48 is cleared. At that stage, the size of theremaining notification image 48 is updated.

Twelfth Exemplary Embodiment

In the above embodiments, the examples are described in which thenotification icon 42 or the notification image 48 displayed by thedisplay unit 33 of the terminal device 3 serves as a trigger forchanging the display image. However, as shown in FIG. 13, a movementbutton 46 for changing the display image can be separately provided, asshown in FIG. 13. There is a case where the occurrence of the conditionchange is notified by the notification image 48 and the notificationicon 42 in the video display area 4. In such a case, by pressing themovement button 46, the image taking direction can be changed to thedirection of the condition change and the image displayed on theterminal device 3 can be changed from the main image to the imageincluding the condition change area.

Thirteenth Exemplary Embodiment

In Japanese Patent Application Laid-Open No. 2004-320286, a cameradevice which has a mechanism that extracts information about apredetermined feature part out of image data and performs zooming to apredetermined area including the predetermined feature part, and acamera device configured to superpose and display a feature part with amarker on a screen are proposed. However, in the above devices, if thetarget feature part does not exist in an image angle of the cameradevice for image taking, it is not possible to extract information aboutthe feature part, and, therefore, zooming and marking is not performed.Accordingly, in order to perform the information extraction, aphotographer has to move the device so that the feature part enters inthe image angle for image taking. Moreover, sometimes the photographerdoes not notice that the feature part exists outside of the image anglefor image taking and fails to capture the feature part. In thethirteenth embodiment, in such a camera device, even if a desiredsubject exists outside of the image angle of the camera device for imagetaking, it can be possible to detect the existence of the subject. Thethirteenth embodiment will be described in detail.

FIG. 14 is a block diagram that illustrates a functional constitution ofa camera device 1400 according to the thirteenth embodiment of thepresent invention. As shown in FIG. 14, the camera device 1400 includesan image forming optical system 1401, an image forming optical systemcontrol unit 1402, an image taking unit 1403, an image taking operationcontrol unit 1404, an image recording unit 1405, an image storage unit1406, an image display unit 1407, a wide range sensor 1408, a subjectcandidate extraction unit 1409, and a subject candidate proposition unit1410.

The image forming optical system 1401 has optical lenses equipped with azoom mechanism. Moreover, the image forming optical system 1401 caninclude a drive mechanism in a pan axis direction and tilt axisdirection.

The image forming optical system control unit 1402, in response to aninstruction from the photographer or based on information from the imagetaking operation control unit 1404 and the subject candidate extractionunit 1409, which will be described below, controls a zoom in the imageforming optical system 1401, or, depending on a configuration of theimage forming optical system 1401, a drive control in a pan directionand a tilt direction.

The image taking unit 1403 is a video input means which includes a videosensor, a sensor signal processing circuit, and a sensor drivingcircuit. As the video sensor, typically, a CCD or a CMOS image sensor isused, and in response to a control signal readout from the sensordriving circuit (not shown), a predetermined video signal (for example,a signal obtained by subsampling, or block readout) is output as imagedata.

The image taking operation control unit 1404 outputs the readout controlsignal to the sensor driving circuit of the image taking unit 1403 andcontrols a timing of actual image taking based on an instruction (imageangle adjustment instruction, depression of shutter, or the like) fromthe photographer, and information from the image forming optical systemcontrol unit 1402 and the subject candidate extraction unit 1409.

The image recording unit 1405 records image data on an external mediumor the like. While, the image storage unit 1406 is composed of asemiconductor memory or the like, temporarily stores the image datatransferred from the image taking unit 1403, and in response to anrequest from the image display unit 1407 and the subject candidateextraction unit 1409, transfers the image data to the image display unit1407 and the subject candidate extraction unit 1409 at a predeterminedtiming.

The image display unit 1407 displays the taken image. Typically, aliquid crystal monitor is used. The image display unit 1407 can be anelectronic viewfinder (EVF). Moreover, the image display unit 1407 caninclude a display unit configured to display a menu in conjunction withan operation unit that enables the photographer to select an imagetaking mode or the like. An image is displayed after a predeterminedprocess is performed to the image data transferred from the imagestorage unit 1406 based on the information of the image forming opticalsystem control unit 1402.

The wide range sensor 1408 performs sensing throughout an area widerthan the area displayed on the image display unit 1407, and the imagesensor which outputs an image signal similar to the image taking unit1403, can be used. Moreover, the wide range sensor 1408 can be a soundreceiving device composed of a sound receiving microphone and anamplifier which amplifies an audio signal. Further, a known sensor suchas a temperature sensor which senses temperature, or an infrared sensor,can be used depending on a subject to be detected. In the thirteenthembodiment, a case in which an image sensor is used will be described.

The subject candidate extraction unit 1409 detects a subject candidatewhich belongs to a specific category out of the output data of the widerange sensor 1408 and the image data output from the image taking unit1403, and extracts information about the subject candidate. The subjectcandidate extraction unit 1409 exists as a program which is executed inan application specific integrated circuits (ASIC), or a processor(reconfigurable processor, DSP, CPU, or the like). It is possible that asubject candidate which belongs to a desired specific category isextracted by registering in advance according to a predeterminedprocedure, and providing a plurality of program modules for detecting aspecific category. Depending on the specific category designated by thephotographer, the program module is dynamically loaded and performed.

The subject candidate proposition unit 1410 presents information aboutthe subject candidate output from the subject candidate extraction unit1409 to the photographer. For example, on the image display unit 1407,information about a position, or size of the subject is highlighted byan arrow. Moreover, using an audio speaker to generate electronic sound,or using audio to present information about a subject candidate to thephotographer is considered. The above-described each block is not alwaysnecessary to be integrally configured as the camera device 1400. Forexample, as described in the first embodiment, each of the blocks can beconnected to a terminal device through a network I/F and the imagedisplay unit 1407 can be included in the terminal device side.

FIG. 15 is a flow chart that illustrates an exemplary processing forproposing a subject candidate in the camera device according to thethirteenth embodiment of the present invention. First, the image displayunit 1407 obtains camera parameters of an image forming optical system,such as a focal length, an image angle, a distortion coefficient, adirection of pan, a direction of tilt from the image forming opticalsystem control unit 1402 or the image taking operation control unit 1404(step S1501). Then, an image is taken by the image taking unit 1403, andthe obtained image data is temporarily stored in the image storage unit1406 (step S1502).

In parallel with the processing at step S1502, sensing is performed bythe wide range sensor 1408 (step S1504). To the image data stored in theimage storage unit 1406 at step S1502, a predetermined image processingis performed in the image display unit 1407 based on the cameraparameters obtained at step S1501 and the image is displayed (stepS1503). The predetermined image processing is, for example, anaberration correction for correcting a distortion in the image formingoptical system 1401, or a processing of converting brightness. Further,a processing of converting a resolution of the image data is performedto fit the resolution of the image display unit 1407. If, for example, aprocessing speed is important, the predetermined image processing may bepartially performed or may not be performed.

Then, a subject candidate extraction processing is performed by thesubject candidate extraction unit 1409 by using the sensing informationtransferred from the wide range sensor 1408 and the image datatransferred from the image storage unit 1406 (step S1505). The subjectcandidate extraction processing will be described below in detail. Ifthere exists a subject candidate (YES at step S1506), the process movesto step S1507. If there is no subject candidate (No at step S1506), theprocess returns to step S1501.

If there exists the subject candidate, as to one of the subjectcandidates, the subject candidate proposition unit 1410 determineswhether the subject candidate is displayed on the image display unit1407, that is, whether the subject candidate is included in the imagestaken by the image taking unit 1403 (step S1507). If the subjectcandidate is displayed on the image display unit 1407, the subjectcandidate proposition unit 1410 highlights the subject candidate by amarker or the like on the image display unit 1407 at step S1508. On theother hand, if the subject candidate is not displayed on the imagedisplay unit 1407, that is, if it is the subject information extractedfrom the information output from the wide range sensor 1408, the processmoves to step S1509 and notifies the photographer of the information ofthe subject candidate. The processing at step S1507 through step S1509is performed to all subject candidates (until it becomes to YES at stepS1510).

Next, the subject candidate extraction processing performed at stepS1505 will be described in detail. FIG. 16 is a block diagram thatillustrates an exemplary configuration of the subject candidateextraction unit 1409. A material detection unit 1621 detects a materialwhich belongs to a predetermined specific category out of the image datainput from the image storage unit 1406 and the data input from the widerange sensor 1408. The size of the detected specific category isvariable. As to the types of the categories, it can be classified, forexample, into a large classification such as a person, a vehicle, amiddle classification in which an older person or a younger person, amale or a female, and the like can be determined among persons, and asmall classification in which a specific person can be detected. Here,for the description, it is supposed that the specific category is a faceof person. A status detection unit 1622 detects a status of an objectdetected by the material detection unit 1621. In a case of a face ofperson, a size, an angle, an age, a sex, an expression of the face, andthe like are detected. An individual recognition unit 1623 performsindividual recognition of the object, in a case of a person, recognizesan individual person.

With reference to a flowchart in FIG. 17, the subject candidateextraction processing procedure will be described. Depending on andirect instruction from the photographer or a mode of the camera device1400 (scenery photographing mode, portrait photographing mode, close-upview photographing mode, distant view photographing mode, or the like),a specific category of a material to be extracted is set (step S1720).Then, the material detection unit 1621 performs material detectionprocessing of the set specific category (step S1721). For example, ifthe set specific category is a specific person, first, face detection isperformed. Then, the status detection unit 1622 performs a statusdetection processing to the detected material of the specific category(step S1722). If the detected material of the specific category is aparson, after the tilt of the face, the size, or the like are detected,further, a shape or a positional relationship of eyes, a nose, a mouthwhich is to be a feature amount to represent a sex, a individual, or anexpression, is detected, and the status of the face is searched.Further, a processing module for individual identification based on thedetected status which is provided in the individual recognition unit1623 is selected (step S1723), and a individual identificationprocessing such as a determination of the face of the individual isperformed (step S1724).

The material detection unit 1621 will next be described in detail. FIG.18 is a block diagram illustrating an exemplary configuration of thematerial detection unit 1621. A feature extraction unit 1831 extracts afeature amount at a predetermined sampling point against input data. Amodel data storage unit 1832 stores in advance information aboutcategories of a plurality of subjects to be extracted as model data. Asdescribed above, if the specific category is a person, model data aboutthe face is stored in the model data storage unit 1832. The model datacan be, for example, face image data, predetermined feature vector dataabout an entire face obtained by a main component analysis or anindependent component analysis, or local feature data about a featurearea on individual face such as eyes, mouth on a face. A matchingprocessing unit 1833 performs a matching processing (similaritycalculation) between the input image data and the model data based onthe extracted feature amount.

As the processing performed in the matching processing unit 1833, forexample, in a case where image data is processed, a method used in animage information extracting apparatus described in Japanese PatentApplication Laid-Open No. 09-130714 can be used. In this imageinformation extracting apparatus, a template model image of a sizecorresponding to a subject distance is generated, while using the imageand scanning a screen. By calculating a normalization correlationfunction at each point, a similarity distribution between a local partof the input image and the model data is calculated. Further, analgorithm based on a spatial position relationship of a local featuredescribed in Japanese Patent No. 3078166 or an algorism based on afolded-neural network circuit described in Japanese Patent ApplicationLaid-Open No. 2002-8032 can be used. Generally, if a maximum value ofthe similarity distribution exceeds a predetermined threshold, a patternwhich belongs to the category is considered to be detected. If the inputdata is audio, for example, a similar predetermined processing using aknown algorism, for example, an algorithm using the Hidden Markov Modelcan be executed.

Further, to reduce a processing time of the material detection, as apre-processing, an operation for narrowing down a search area can beperformed in which a extraction of a primary feature amount (low-levelfeature amount such as movement vector, color information) out oftime-series input image data is performed by using a known method. Acandidate area having a higher possibility that a main subject existstherein, is extracted, and the narrowing down of search area can beperformed. For example, based on color information, an image area havinga color close to a color of a important part (for example, skin color ofa person) of model image data which is registered in advance isextracted by a threshold processing, and the narrowing down operationcan be performed. Further, a area having a movement vector amountgreater than a predetermined size or a detection area similar to a headpart of a person can be extracted as a subject candidate area. Whenusing the movement vector, it is preferable that the image taking unit1403 is fixed and placed at a predetermined position. In a case ofhand-held image taking, a global movement vector amount (Ego-movementvector) generated in connection with a movement of the image taking unit1403 is extracted. After the global movement vector amount is offsetfrom an entire global movement vector distribution, an area split basedon the global movement vector amount is performed. Thus, a candidatearea of a moving member such as a person can be computed.

It is further noted that the processing performed in the matchingprocessing unit 1833 according to this embodiment is not limited to theabove-described processing method, and that known methods can beappropriately applied.

Next, the status detection unit 1622 (from FIG. 16) will be described.The status detection unit 1622 detects a status class about a specificcategory (face etc.) designated by the photographer or a mode of thecamera device 1400 according to this embodiment. In the status class, acategory designated by the photographer or a mode of the camera device1400 according to this embodiment is a large classification, and statusdetection is performed by using a middle classification level, and asmall classification level. The status category is, for example, in acase of a person such as this embodiment, a facial expression and anorientation of face, classified in advance depending on a material to bedetected and stored in a predetermined memory.

In FIG. 19, an example of the status class classified in a treestructure per each class of a material to be stored in the memory isshown. As shown in FIG. 19, if the large classification is “person”, andthe middle classification is “orientation of face”, as the smallclassification, classes segmented in a pan direction and a tiltdirection are determined in advance. If a status class of a face isdetected, first, detection of feature parts effective for the detectionof a sex, a facial expression, and an orientation of face such as anouter corner of the eye, mouth area is performed. As a result, as thestatus class, a category such as “slanting, male, no expression” isdetected.

Next, the individual recognition unit 1623 (from FIG. 19) will bedescribed. The individual recognition unit 1623 performs an individualidentification processing in the status class detected in the statusdetection unit 1622 among the area of face detected in the materialdetection unit 1621. FIG. 20 illustrates a configuration of theindividual recognition unit 1623. In this case, the status class isclassified in advance as a tree structure having a large classification,a middle classification, and a small classification, and a processingmodule group is arrayed for each large classification. For example, alarge classification indicates “face”, a middle classification indicatesan orientation of face such as “front”, and a small classificationindicates a status such as “age”. The classification style of the statusclass can be a style other than the tree structure. A control unit forselecting an individual identification processing module specialized inthe status class detected by the status detection unit 1622, and theindividual identification processing module groups are provided in theindividual identification unit 1623. The control unit and the individualidentification processing module groups can be stored in advance as aprogram in a predetermined memory or as a hard IP core stored in asystem LSI in the subject candidate extraction unit 1409.

For example, if it is detected by the status detection unit 1622 thatthe state of a face belongs to the status class of “front, male”, theindividual identification unit 1623 selects an individual identificationmodule corresponding to the status class. The individual identificationmodule corresponding to each status class has a certain width ofidentification performance. That is, even if the individualidentification module is specialized in “front, male”, within anattribution range such as a certain orientation of face or relativelywide range of age (for example, male faces from teens to seventies), theindividual identification module is generated by learning in advance sothat robust identification against variation is possible.

As such identification module, for example, a support vector machineconfigured to identify faces of two specific persons in a specificstatus category is generated by learning and used. For the learning, forexample, a face data set about two specific persons in a certain rangearound a specific status category (in this example, front, smile) isused to train the support vector machine to identify the two person. Asan example of the range around a specific status category, face turningwithin the range of ±20 degrees from front, and faces of male of variousages are included. A support vector machine of a “one-against-all” typecan be generated. In this case, the support vector machine can begenerated to the number of individuals that should be identified.

In this case, as to the status class in the specific category, suchsupport vector machines which perform an identification of binary aregenerated by learning up to the number of combinations of all pairs.When performing the identification process, it is not necessary to carryout the identification process as to the combinations of all pairs, butif a binary tree system is used, it is enough to carry out a comparisonof the number of classes—1. This example is discussed in a document,Guo, Li, and Chan “Face Recognition by Support Vector Machines, in Proc.of 14th International Conf. on Automatic Face and Gesture Recognition,2000. However, sometimes it is difficult to perform such generation ofthe identification module in advance as to all status classes of allpairs.

Accordingly, the generation of the identification module can beperformed as a result of a detection of the status class, as to a personwho could not be identified, after the identification and the result isto be accumulated. For example, if a detected specific status categorycannot be identified (or, the reliability of the identification resultis assumed to be low), the generation of identification module can betried by a method of learning by using face image data of the sameperson in the status class.

As each module of the individual identification processing, other thanthe above method based on the support vector machine, a method using ahierarchical neural network or a method using a statistical patternrecognition can be used. For example, a method using a folded neuralnetwork as the hierarchical neural network can be used as a base(document: Lawrence et al., “Face Recognition: A ConvolutionalNeural-Network Approach, IEEE Transactions on Neural Networks, vol. 8,pp. 98-113, 1997). In this case, face image data for learning isprovided for each status class, and the individual identificationprocessing module can be generated by performing learning with ateacher. As described, it is possible to configure each module by usingthe various methods.

As described above, first, a status class (orientation of face, sex, andthe like) is determined both at a time of identifying the face, and atlearning, and based on the result, an identification module specializedin a specific status class is generated. Otherwise, an identificationoperation of an individual is performed by selectively starting alearned specific identification module (it is not always to be oneidentification module). As a method for the selective starting, based onthe detected status class, a start-up of an individual identifyingmodule corresponding to a class near the detected status class can beperformed.

As described above, by performing the determination (relatively lightprocessing) to which status class (sex, orientation of face, or thelike) of types set in advance belongs before the individualidentification, it is possible to effectively narrow down a search areafor individual identification in a feature space. After thedetermination, by performing an identification operation by anidentification module generated by being specified to the status classdetected, it is possible to effectively and accurately identify ascompared with the case where the individual identification is performedin a set which includes all status classes.

Next, the subject candidate proposition unit 1410 (from FIG. 14) will bedescribed. The subject candidate proposition unit 1410 determineswhether the subject candidate extracted by the subject candidateextraction unit 1409 is in the image display unit 1407 or out of theimage display unit 1407 (step S7 in FIG. 15). If it is determined thatthe subject candidate is in the image display unit 1407, the subjectcandidate proposition unit 1410 proposes the subject candidate byhighlighting by maker (step S8).

FIG. 21 illustrates an example in which information about face positionsof persons who are subject candidates on the image display unit 1407 ispresented by the subject candidate proposition unit 1410. In the exampleshown in FIG. 21, while in the faces of persons who are subjectcandidates, maker (frame) displays 2131 and 2132 are displayed, it ispossible to display, for example, attributes (sex, age, expression,individual name) of the person. It is also possible to display theseattributes not by character strings but by abstract attributes of iconsand the like. If a plurality of subject candidates is detected bysubject candidate extraction unit 1409, as shown in FIG. 21, it ispossible to display them as they are by superposing them, or controlthem as described below. That is, if the level of importance of thesubject candidates is determined by an instruction from the photographerand the photographing mode, or if the level of importance is calculatedfrom a past history of the subject candidate extraction result, it ispossible to selectively propose the subject candidate.

FIG. 22 illustrates a case in which among the plurality of subjectcandidates, if at least a part of them are outside of the image displayunit 1407, the subject candidate proposition unit 1410 displaysinformation display (step S1509 in FIG. 15). In this case, with respectto the subject candidate which is out of the image display unit 1407, anarrow indication 2242 indicates its general information about how farthe subject candidate is away from the image display area and a messageindication 2243 presents more detailed information. By using theinformation, it is possible to automatically change a pan angle or atilt angle, or a image angle of the image forming optical system 1401 sothat the image taking unit 1403 is able to take image of the subjectcandidate.

As described above, if the subject candidate is out of the image takingimage angle, it is possible to detect the existence of the subjectcandidate. Further, it is possible to present the information about theposition of the subject candidate which exists out of the image takingimage angle.

In the above thirteenth embodiment, based on the image data from theimage taking unit 1403 or the information form the wide range sensor1408, the subject candidate is detected. However, if the informationobtained from the wide range sensor 1408 includes the image taking areaof the image taking unit 1403, it is possible to detect the subjectcandidate based on the information from the wide range sensor 1408without using the image data from the image taking unit 1403.

Further, in the thirteenth embodiment, the wide range sensor 1408outputs the image data. However, a category can be set depending on asubject to be detected, for example, for a temperature sensor, acategory of temperature is set, and for an sound receiving device, acategory of audio is set. Particularly, in a case of the temperaturesensor, for example, a predetermined temperature range of 35 degrees to37 degrees is set and in a case of the sound receiving device, forexample, a category to be detected is set to “laughter” or “sound equalto or more than XX decibel and the like. Further, a determinationwhether the subject is in the image taking area of the image taking unit1403 can be made based on a image angle of the image taking angle 1403.

Alternative Modification of the Thirteenth Embodiment

In a thirteenth embodiment, while the image taking unit 1403 and thewide range sensor 1408 are independently configured respectively, it ispossible that the image taking unit 1403 and the wide range sensor 1408are integrally configured. In this modification, this case will bedescribed. The basic configuration and processing of a camera device aresimilar to those described in the above-described thirteenth embodiment,and therefore, only potions different from the thirteenth embodimentwill be described.

FIG. 23 illustrates a relationship between the image taking unit 1403and the wide range sensor 1408 which are integrally configured by anarea sensor which includes a sensing area of the wide range sensor 1408.Accordingly, image data output from a part of the area sensorcorresponds to the image data output from the image taking unit 1403.The image data output from the area of the whole of the sensor or thearea other than the area corresponding to the image taking unit 1403,corresponds to the image data output from the wide range sensor 1408.

A size of the image taking unit 1403 is varied in response to a imageangle instruction from the photographer. If an instruction is given tonarrow the image angle (zoom in), for example, image data in an areaindicated by dotted lines that is narrower than the area indicated bysolid lines in FIG. 23 is output as image data of the image taking unit1403. On the other hand, if an instruction is given to widen the imageangle (zoom out), for example, image data in an area indicated by dottedlines that is wider than the area indicated by solid lines in FIG. 23 isoutput as image data of the image taking unit 1403. In consideration ofresolutions, the area of the image taking unit 1403 is adjusted so thatthe area is within an area of a fixed rate against the wide range sensor1408 by adjusting an image angle of the image forming optical system 1.

Hereinafter, an area around the center of the area sensor is referred toas the image taking unit 1403, and, the whole area or an area other thanthe area corresponding to the image taking unit 1403 is referred to asthe wide range sensor 1408. Image data output from the image taking unit1403 is to be an image displayed on the image display unit image displayunit 1407.

With reference to FIG. 24, an exemplary image taking processingperformed at step S1502 in FIG. 15 will now be described if the imagetaking unit 1403 and the wide range sensor 1408 are the same device. Ifan image taking instruction signal is generated by a depression of ashutter (not shown) by a photographer step S2431), a current image angleA1 is computed based on camera parameters in the image forming opticalsystem 1401 (step S2432). This current image angle A1 corresponds to theangle of the wide range sensor 1408. Then, an image angle A2 of theimage taking unit 1403 is computed based on camera parameters designatedby the photographer (step S2433). Then, it is determined whether thecomputed image angle A2 is in a predetermined range against the currentimage angle A1 (step S2434). If it is determined that the image angle A2is not in the predetermined range, an image angle of the image formingoptical system 1401 is adjusted by the image forming optical systemcontrol unit 1402 so that the image angle A2 falls within thepredetermined range (step S2435). Then, an actual image takinginstruction signal is sent to the image taking unit 1403, and imagetaking is performed (step S2436). If the image angle A2 falls within thepredetermined range against the image angle A1 (YES at step S2434), theprocess moves to step A2436 and an image taking is performed. Then theimage data obtained by the image taking is temporarily stored in theimage storage unit 1406 (S2437).

With reference to FIG. 25, exemplary image processing performed at stepS1503 in FIG. 15 according to this modification will be brieflydescribed. An image angle A1 of the image forming optical system 1401 iscomputed based on camera parameters obtained from the image formingoptical system control unit 1402 (step S2541). As described above, thisimage angle A1 corresponds to the angle of the wide range sensor 1408.Then, an image angle A2 designated by the photographer is computed (stepS2542).

Then, based on the image angles A1 and A2, an image is cut out to be theimage of the image angle A2 out of the image data obtained from the areasensor, and a resolution conversion is performed (step S2543). After apredetermined image processing such as an aberration correction forcorrecting a distortion in the image forming optical system 1401, or aprocessing of converting brightness, is performed (step S2544), theimage is displayed on the image display unit 1407 (step S2545). Thepredetermined image processing is, for example, an aberration correctionfor correcting a distortion in the image forming optical system 1401, ora processing of converting brightness. Further, a resolution of theimage data is converted to be suitable for the resolution of the imagedisplay unit 1407. The predetermined image processing may not bepartially or completely performed if a processing speed is important,for example.

As described above, according to this modification, the image takingunit 1403 and the wide range sensor 1408 can be realized as the sameconfiguration. In that case, as compared to the case in which the imagetaking unit 1403 and the wide range sensor 1408 are configuredseparately, the configuration can be simplified.

Fourteenth Exemplary Embodiment

In the fourteenth embodiment, in addition to the configuration in thethirteenth embodiment, further an exemplary moving member anticipationunit 2611 (see FIG. 26) is included, which will be described.

FIG. 26 is a block diagram illustrating an exemplary functionalconfiguration of a camera device 2600 according to the fourteenthembodiment of the present invention. The configuration shown in FIG. 26is further provided with a moving member anticipation unit 2611 inaddition to the configuration of FIG. 13, as compared to theconfiguration of FIG. 14 described in the thirteenth embodiment. Themoving member anticipation unit 2611 anticipates time that a subjectcandidate which is out of an image taking image angle enters in theimage taking image angle, and time that a subject candidate which is inan image taking image angle moves out of the image taking image angle.The subject candidate proposition unit 1410 presents the anticipation tothe photographer as information. Other constituent elements are similarto those in FIG. 14, and therefore, the same reference numerals arenumbered and their description is omitted.

The subject candidate extraction unit 1409, as well as the subjectcandidate extraction unit 1409 in the thirteenth embodiment, has thematerial detection unit 1621 which detects multi-category such as eachpart of body of a person, a vehicle, or limited categories registered inadvance. Further, in the fourteenth embodiment, if the material detectedobject is a person, the status detection unit 1622 which detectsattribution information such as an age or a sex has a function ofdetecting a moving state. As a check class model data for determiningwhether the subject is moving, data which is time-series data ofmaterial detection result, and variation pattern of a feature pointposition of each part of a moving member are used. The feature pointposition of each part of a moving member is in a four-dimensional spacein which a time axis is added to a three-dimensional space, or athree-dimensional space in which a time axis is added to atwo-dimensional space projected to an image taking sensor plane.

FIG. 27 is a flowchart illustrating an exemplary processing forpresenting a subject candidate according to the fourteenth embodiment ofthe present invention. The processing shown in FIG. 27 is similar tothose in FIG. 15 which is described in the thirteenth embodiment untilthe processing at step S1506 that determines whether a subject candidateexists. Therefore, their descriptions are omitted and processingperformed after a determination that the subject candidate exists willbe described.

At step S1506, if it is determined that a subject candidate exists, theprocess moves to step S2751 and a moving member extraction processing isperformed. As described above, the processing is performed by labelingin advance a subject candidate and a material in moving state accordingto a result of a status detection processing (step S1722 in FIG. 17),and checking the label. The moving member extraction processing will bedescribed in detail below. Then, it is determined whether the subjectcandidate is a moving member (step S2752). If it is determined that thesubject candidate is not a moving member, processing similar to that insteps S1507 through S1509 in FIG. 15 is performed and the process movesto step S2756.

On the other hand, if the subject candidate is a moving member (YES atstep S2752), it is determined whether the subject candidate is displayedon the image display unit 1407, that is, the subject candidate isincluded in images taken by the image taking unit 1403 or not (stepS2753). If the subject candidate is displayed on the image display unit1407, the subject candidate proposition unit 1410 performs a markerindication to the subject candidate, and further, presents thephotographer information obtained by the moving member anticipation unit2611 by outputting a message on the image display unit 1407 or byoutputting sound from a speaker (step S2754). The information obtainedby the moving member anticipation unit 2611 is, for example, a movingspeed, time moving from the interior of a display screen to the outside,or, a moving speed of a camera device, an exposure, a focus control tobe most suitable instruction information to perform so-called panningphotography which performs image taking of a moving member withoutblurring.

On the other hand, if the subject candidate is not displayed on theimage display unit 1407, that is, in a case where subject candidateinformation is extracted from only information output from the widerange sensor 1408 (NO at step S2753), information obtained by the movingmember anticipation unit 2611 is displayed on the image display unit1407. In this case, for example, other than a category or an attributionof a subject, information about anticipated time the subject candidateenters an image taking image angle, or an instruction, shutter timing, afocus, an exposure most suitable for performing above-described panningphotography is presented.

Processing after step S2752 is performed to all subject candidates(until it becomes YES at step S2756), and the processing is completed.

With reference to FIG. 28, the moving member anticipation unit 2611 willbe described. FIG. 28 is a block diagram illustrating an exemplaryconfiguration of the moving member anticipation unit 2611. In order toextract a movement vector, a frame buffer 2841, temporarily stores theimage data stored in the image storage unit 1406, information obtainedfrom the wide range sensor 1408, and a result of the subject candidateextraction unit 1409. A movement vector extraction unit 2842 obtains amovement vector of a subject candidate which is a moving member by usingimage data obtained from the frame buffer 2841, the information obtainedfrom the wide range sensor 1408, and the time-series data which is theresult of the subject candidate extraction unit 1409. A moving memberdetermination unit 2843 computes a speed of a moving member extracted inmovement vector extraction unit 2842 and outputs the speed astime-series data. A moving member position anticipating unit 2844anticipates a position of a moving member from the time-series data ofspeed of the moving member obtained from the moving member determinationunit 2843, and computes a time when the subject candidate comes to aboundary of an image angle of the image taking unit 1403.

With reference to a flowchart in FIG. 29, the processing in the movingmember anticipation unit 2611 will be described. The processing isperformed at step S2754 and step S2755. First, time-series data of animage is obtained from the frame buffer 2841 (step S2960). Then, amovement vector is extracted from the time-series image data (stepS2961). In this case, the moving vector can be obtained by using knowntechniques, for example, an optical flow. Even if the camera device 2600is moving, by removing a global movement vector, only a movement vectorof a moving member to which an attention is given, can be extracted.Then, a camera parameter is obtained from the image forming opticalsystem control unit 1402 (step S2962). Since a focal length of the imageforming optical system 1401 is obtained from the camera parameter andthe size of the moving member is known in advance by the subjectcandidate extraction unit 1409, a distance to the moving member iscomputed and a moving speed of the moving member is anticipated fromboth elements, and a distribution of the movement vector (step S2963).Then, from the anticipated moving speed, a positional relationshipbetween the image display unit 1407 and the display area is obtained intime-series (step S2964). At the same time, a speed of the camera device2600 moving with the moving member which is most suitable for so-calledpanning photography, can be obtained from the camera parameter. In thiscase, an image taking is performed by focusing only a moving member.Time the moving member arrives to the boundary between the image displayunit 1407 is presented as an anticipated time of framing in or framingout (step S2965).

FIG. 30 illustrates an example of a display of a result (YES at step2754) by the moving member anticipation unit 2611 in a case where asubject candidate is a moving member (YES at step S2752) and the subjectcandidate is in a display area of the image display unit 1407 (YES atstep S2753). In this case, the moving member is a person, and the personis moving from the right side to the left side in the screen. By thesubject candidate proposition unit 1410, a marker (frame) is displayedon the face of the person where a direction of movement is indicatedtogether with an arrow in a moving direction. Further, an anticipatedtime when the person is framed-out is displayed as a message.

FIG. 31 illustrates an example of a display of a result (YES at step2755) by the moving member anticipation unit 2611 in a case where asubject candidate is a moving member (YES at step S2752) and the subjectcandidate is out of a display area of the image display unit 1407 (NO atstep S2753). In this case, the moving member is a specific person A, anda case is assumed in which the person enters into the screen from theright outside of the display screen. A moving speed is presented with anarrow directing a direction of movement, and the name of the person andtime of frame-in is displayed as a message. Further, a moving speed ofthe camera device most suitable for panning photography is displayed.

As described above, according to the fourteenth embodiment of thepresent invention, in addition to effects similar to the abovethirteenth embodiment, if the subject candidate is a moving member,further detailed information can be proposed.

In the first embodiment and the fourteenth embodiment, the markerindication which indicates a subject candidate extracted by the subjectcandidate extraction unit 1409 is performed on the image display unit1407. However, in the present invention, it is not always necessary toperform the marker indication. By using the extracted result of thesubject candidate which is out of the image taking area of the imagetaking unit 1403, various controls can be performed, for example, basedon the extracted information of the subject candidate, a camera deviceis controlled so that the subject candidate is taken by the image takingunit 1403.

Other Exemplary Embodiments, Features and Aspects of the PresentInvention

The present invention can also be achieved by providing the system orthe device with a storage medium which records a program code ofsoftware implementing the function of the embodiment and by reading andexecuting the program code stored in the storage medium with a computerof the system or the device (the CPU or the MPU).

In this case, the program code itself, which is read from the storagemedium, implements the function of the embodiment mentioned above, andaccordingly, the storage medium storing the program code constitutes thepresent invention.

As the storage medium for supplying such program code, a flexible disk,a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R,a magnetic tape, a nonvolatile memory card, a ROM, and the like can beused.

In addition, the function according to the embodiments described aboveis implemented not only by executing the program code read by thecomputer, but also implemented by the processing in which an OS(operating system) or the like carries out a part of or the whole of theactual processing on the basis of the instruction given by the programcode.

Further, in another aspect of the embodiment of the present invention,after the program code read from the storage medium is written in afunction enhancing board inserted in the computer or a memory which isprovided in a function enhancing unit connected to the computer, the CPUand the like provided in the function enhancing board or the functionenhancing unit carries out a part of or the whole of the processing toimplement the function of the embodiment as described above.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application Nos.2005-138367 filed May 11, 2005 and 2005-256862 filed Sep. 5, 2005, whichare hereby incorporated by reference herein in its entirety.

1. A network camera system configured to display an image picked up byan image pickup apparatus on a terminal device via a network, the systemcomprising: a detection unit configured to detect an occurrence of acondition change of a first type based on notification from a firstsensor and to detect an occurrence of a condition change of a secondtype based on notification from a second sensor; a notification unitconfigured to notify, in a case where the detection unit detects theoccurrence of the condition change of the first type, the terminaldevice of a direction of the condition change of the first type suchthat a first icon corresponding to the first sensor and having a shapethat can specify the direction of the condition change of the first typeis displayed on a display of the terminal device, and to notify, in acase where the detection unit detects the occurrence of the conditionchange of the second type, the terminal device of a direction of thecondition change of the second type such that a second iconcorresponding to the second sensor and having a shape that can specifythe direction of the condition change of the second type is displayed onthe display of the terminal device; and a changing unit configured tochange a pickup direction of the image pickup apparatus to the directionof the condition change of the first type or the direction of thecondition change of the second type in accordance with a selection ofthe first icon or the second icon displayed on the display of theterminal device.
 2. The network camera system according to claim 1,wherein the terminal device displays the first icon or the second iconin a superposed manner in a direction of the condition change relativeto a center of a video display area of the terminal device.
 3. Thenetwork camera system according to claim 2, wherein the first icon orthe second icon is not displayed in the superposed manner if an image ofthe condition change area is included in a video display area of theterminal device.
 4. The network camera system according to claim 2,wherein in displaying the first icon or the second icon in thesuperposed manner, a size of the icon can be changed.
 5. The networkcamera system according to claim 1, wherein the changing unit changes apickup direction of the image pickup apparatus to the direction of thecondition change of the first type in accordance with the selecting thefirst icon and changes the pickup direction of the image pickupapparatus to the direction of the condition change of the second type inaccordance with the selecting the second icon.
 6. The network camerasystem according to claim 1, wherein a plurality of icons is displayedin a superposed manner in a number of the condition changes.
 7. Thenetwork camera system according to claim 1, wherein the terminal devicedisplays an image picked up after changing the direction information onthe terminal device via the network, and wherein the detection unitdetects the occurrence of the condition change.
 8. The network camerasystem according to claim 1, wherein the first sensor is one of anoptical sensor, a voice sensor, and a temperature sensor or acombination thereof.
 9. A method for controlling a network camera systemconfigured to display an image picked up by an image pickup apparatus ona terminal device via a network, the method comprising: detecting anoccurrence of a condition change of a first type based on notificationfrom a first sensor and detecting an occurrence of a condition change ofa second type based on notification from a second sensor; notifying, ina case where the occurrence of the condition change of the first type isdetected in the detecting step, the terminal device of a direction ofthe condition change of the first type such that a first iconcorresponding to the first sensor and having a shape that can specifythe direction of the condition change of the first type is displayed ona display of the terminal device, and notifying, in a case where theoccurrence of the condition change of the second type is detected in thedetecting step, the terminal device of a direction of the conditionchange of the second type such that a second icon corresponding to thesecond sensor and having a shape that can specify the direction of thecondition change of the second type is displayed on the display of theterminal device; and changing a pickup direction of the image pickupapparatus to the direction of the condition change of the first type orthe direction of the condition change of the second type in accordancewith a selection of the first icon or the second icon displayed on thedisplay of the terminal device.
 10. An image pickup apparatus configuredto send a picked up image to a terminal device via a network, theapparatus comprising: a detection unit configured to detect anoccurrence of a condition change of a first type based on notificationfrom a first sensor and to detect an occurrence of a condition change ofa second type based on notification from a second sensor; a notificationunit configured to notify, in a case where the detection unit detectsthe occurrence of the condition change of the first type, the terminaldevice of a direction of the condition change of the first type or theinformation for obtaining the direction of the condition change of thefirst type such that a first icon corresponding to the first sensor andhaving a shape that can specify the direction of the condition change ofthe first type is displayed on a display of the terminal device, and, tonotify, in a case where the detection unit detects the occurrence of thecondition change of the second type, the terminal device of a directionof the condition change of the second type or the information forobtaining the direction of the condition change of the second type suchthat a second icon corresponding to the second sensor and having a shapethat can specify the direction of the condition change of the secondtype is displayed on the display of the terminal device; a receivingunit configured to receive information relative to either of the firsticon or the second icon which is selected in the terminal device; and achanging unit configured to change a pickup direction of the imagepickup apparatus to the direction of the condition change of the firsttype or the direction of the condition change of the second type inaccordance with the receiving the information relative to the icon whichis selected in the terminal device.
 11. A terminal device configured toreceive and display an image picked up by an image pickup apparatus viaa network, the device comprising: an obtaining unit configured to obtaina direction of a condition change of a first type and to obtain adirection of a condition change of a second type based on notificationfrom the image pickup apparatus; a display unit configured to display ona display, in a case where the obtaining unit obtains the direction ofthe condition change of the first type corresponding to a first sensor,a first icon corresponding to the first sensor and having a shape thatcan specify the direction of the condition change of the first type, andto display on the display, in a case where the obtaining unit obtainsthe direction of the condition change of the second type correspondingto a second sensor, a second icon corresponding to the second sensor andhaving a shape that can specify the direction of the condition change ofthe second type; and a transmission unit configured to transmitinformation to the image pickup apparatus, in response to a selection ofthe first icon or the second icon, for changing a pickup direction ofthe image pickup apparatus to the direction of the condition change ofthe first type or the direction of the condition change of the secondtype.
 12. A non-transitory computer readable medium containingcomputer-executable instructions, the medium utilizable in an imagepickup apparatus configured to send a picked up image to a terminaldevice via a network, the medium comprising: computer-executableinstructions for detecting an occurrence of the condition change of afirst type based on notification from a first sensor and for detectingan occurrence of a condition change of a second type based onnotification from a second sensor; computer-executable instructions fornotifying, in a case where the occurrence of the condition change of thefirst type is detected in the detecting step, the terminal device of adirection of the condition change of the first type or the informationfor obtaining the direction of the condition change of the first typesuch that a first icon corresponding to the first sensor and having ashape that can specify the direction of the condition change of thefirst type is displayed on a display of the terminal device, and, fornotifying, in a case where the occurrence of the condition change of thesecond type is detected in the detecting step, the terminal device of adirection of the condition change of the second type or the informationfor obtaining the direction of the condition change of the second typesuch that a second icon corresponding to the second sensor and having ashape that can specify the direction of the condition change of thesecond type is displayed on the display; computer-executableinstructions for receiving information relative to the first icon or thesecond icon which is selected in the terminal device; andcomputer-executable instructions for changing a pickup direction of theimage pickup apparatus to the direction of the condition change of thefirst type or the direction of the condition change of the second typein accordance with the receiving the information relative to the iconwhich is selected in the terminal device.
 13. A non-transitory computerreadable medium containing computer-executable instructions, the mediumutilizable in a terminal device configured to receive and display animage picked up by an image pickup apparatus via a network, the mediumcomprising: computer-executable instructions for obtaining a directionof a condition change of a first type and for obtaining a direction of acondition change of a second type based on notification from the imagepickup apparatus; computer-executable instructions for displaying on adisplay, in a case where the direction of the condition change of afirst type corresponding to the first sensor is obtained in theobtaining step, a first icon corresponding to the first sensor andhaving a shape that can specify the direction of the condition change ofthe first type, and for displaying on the display, in a case where thedirection of the condition change of the second type corresponding to asecond sensor is obtained in the obtaining step, a second iconcorresponding to the second sensor and having a shape that can specifythe direction of the condition change of the second type; andcomputer-executable instructions for transmitting information to theimage pickup apparatus, in response to a selection of the first icon orthe second icon, for changing a pickup direction of the image pickupapparatus to the direction of the condition change of the first type orthe direction of the condition change of the second type.